The temporal MoCo model thus tries to learn similar embeddings for neighboring12 frames. Intheexemplarsplit,weuse27ofthese16 exemplars totrain linear classifiers ontopofthepre-trained and frozen features. We will do a better job of23 motivating this important question intherevised paper. Wehaverevised the paper to acknowledge and cite this earlier work.(4)44 Mostvideo47 datasets incomputer vision consist ofrelatively short video clips instead (typically on the order oftens ofseconds);48 in this setting, the temporal classification model becomes similar to a video instance embedding model, which has49 beenexploredbefore(cf.

We thank all four reviewers for their constructive comments. We respond to each reviewer's comments separately below. We also report some results from new experiments suggested by the reviewers. This idea is illustrated in Figure 2 in the paper. Second, we followed the reviewer's suggestion and developed a temporal version of MoCo as a new baseline: MoCo-Img: 46.6% in the labeled S dataset).